WHOI-2006-01 


Woods  Hole  Oceanographic  Institution 


1930 


FISH_ROCK:  A  Tool  for  Identifying  and  Counting  Benthic 
Organisms  in  Bottom  Photographs 


by 

Vicki  Lynn  Ferrini  and  Hanumant  Singh 


Woods  Hole  Oceanographic  Institution 
Woods  Hole.  MA  02543 


January  2006 


Technical  Report 


Funding  was  provided  by  the  National  Oceanic  and  Atmospheric  Administration 
under  Grant  No.  AB133F05SE5828. 

Approved  for  public  release;  distribution  unlimited. 


WHOI-2006-01 


FISH_ROCK:  A  Tool  for  Identifying  and  Counting  Benthic  Organisms 

in  Bottom  Photographs 


by 

Vicki  Lynn  Ferrini  and  Hanumant  Singh 


Woods  Hole  Oceanographic  Institution 
Woods  Hole,  Massachusetts  02543 


January  2006 

Technical  Report 


Funding  was  provided  by  the  National  Oceanic  and  Atmospheric  Administration  under  Grant 

No.  AB133F05SE5S28. 


Reproduction  in  whole  or  in  part  is  permitted  for  any  purpose  of  the  United  States 
Government.  This  report  should  be  cited  as  Woods  Hole  Oceanog.  Inst.  Tech.  Rept., 

WHOI-2006-01. 


Approved  for  public  release;  distribution  unlimited. 


Approved  for  Distribution: 


Table  of  Contents 


List  of  Figures . 3 

1 .0  Introduction . 4 

2.0  Data  Input . 6 

2. 1  linage  input . 6 

2.2  Navigation  data  input . 7 

2.2.1  Use  of  the  cat_nav.mat  file  generated  for  SeaBED . 7 

2.2.2  Creating  a  cat_nav.mat  file . 7 

2.2.3  Using  a  text  file  for  nav  input . 8 

3.0  Using  F1SH  ROCK . 10 

3.1  Choose  Start  Image . 10 

3.2  Nav  Display  -  Image  Data . 1 1 

3.3  Navigating  Through  a  Dataset . 12 

3.4  Database  Tools . 13 

3.4.1  Selecting  Organism  Type/Name . 13 

3.4.2  Counting  Organisms . 13 

3.4.1  Annotation  of  Identified  Organisms . 14 

3.4.1  Resetting  data .  14 

3.5  Measurement  Tools .  15 

3.5.1  Length  Measurement . 15 

3.5.2  Area  Measurement . 17 

3.6  Substrate  Type .  19 

3.7  Other  Tools . 20 

3.7.1  Grid  On/Off. . 20 

3.7.1  Zoom  In/Out . 21 

3.7.2  Enable/Disable  Pan . 2 1 

3.7.4  Save  Annotated  Tif . 21 

3.9  Closing  the  GUI . 23 

4.0  Database  Output  -  FCT  Philosophy . 24 

5.0  Conclusions  and  Plans  for  Future  Development . 25 

6.0  References . 27 

7.0  Acknowledgements . 27 


2 


List  of  Figures 


Figure  1:  SeaBED  AUV . 4 

Figure  2:  Setting  look  angles  in  FISHROCK . 6 

Figure  3:  Example  pxlflyNAV.txt  file  that  is  generated  by  FISH  ROCK . 7 

Figure  4:  Example  of  text  file  used  as  navigational  input . 9 

Figure  5:  FISH  ROCK  Graphical  User  Interface  (GUI)  as  it  appears  on  startup . 10 

Figure  6:  Choose  Start  Image  browsing  tool .  1 1 

Figure  7:  FISH  ROCK  as  it  appears  after  choosing  the  first  image  to  be  analyzed .  1  1 

Figure  8:  FISH_ROCK  as  it  appears  after  advancing  to  second  image .  12 

Figure  9:  Menu  window  that  appears  to  confirm  saving  data .  13 

Figure  10:  Database  Tools  portion  of  FISH  ROCK  GUI .  13 

Figure  1  1:  FISH  ROCK  with  three  rockfish  identified  and  marked .  14 

Figure  12:  Measurement  Tools  portion  of  FISH  ROCK  GUI . 15 

Figure  13:  Example  of  Measurement  Tools  in  FISH  ROCK  GUI . 16 

Figure  14a:  Assign  length  to  organisms  menu .  16 

Figure  14b:  List  of  organisms  to  assign  length . 16 

Figure  15:  FISH  ROCK  after  assigning  a  length  measurement  to  an  organism .  17 

Figure  16:  Use  of  area  measurement  tool .  18 

Figure  17:  Change  of  symbol  after  assigning  the  area  measurement . 19 

Figure  18:  Substrate  Type  tool . 19 

Figure  19:  Benthic  Habitat  Binary  Classification  Scheme . 20 

Figure  20:  Example  of  pressing  Grid  On . 21 

Figure  2 1  a:  Example  of  annotated  image  in  FISHROCK . 22 

Figure  2  I  a:  Example  of  annotated  tif  file  created  by  FISH  ROCK . 22 

Figure  22:  Photo-mosaic  of  the  three  adjacent  images  presented  in  this  report . 26 


3 


1.0  Introduction 

Recent  advances  in  underwater  robotics  and  imaging  technology  now  enable  the 
rapid  acquisition  of  large  datasets  of  near-bottom  high-resolution  digital  imagery.  These 
images  provide  the  potential  for  developing  a  non-invasive  technique  for  fisheries  data 
acquisition  that  reveals  the  organisms  in  their  natural  habitat  and  can  be  used  to  identify 
important  habitat  characteristics.  Using  these  large  datasets  effectively,  however, 
requires  the  development  of  computer-based  techniques  that  increase  the  efficiency  of 
data  analysis. 

FISH  ROCK  is  a  graphical  user  interface  (GUI)  that  was  designed  in  Matlab  as  a 
database  building  tool  for  identifying  and  counting  benthic  organisms  and  for 
categorizing  the  substrate  in  bottom  photographs  collected  with  the  SeaBED  Autonomous 
Underwater  Vehicle  (AUV;  Figure  1).  It  was  initially  developed  for  researchers  from  the 
NOAA  Pacific  Marine  Fisheries  Division  seeking  to  employ  new  tools  for  studying  and 
assessing  the  population,  abundance,  and  distribution  of  benthic  organisms  relevant  to 
fisheries  management  issues.  During  a  2-week  research  cruise  aboard  the  RJV  Thompson 
(TN  1 86;  Chief  Scientist:  Elizabeth  Clarke)  over  30,000  digital  images  were  collected  of 
the  seafloor  using  a  pixelfly  high  dynamic  range  camera  mounted  on  the  SeaBED  AUV. 
FISH  ROCK  was  created  to  facilitate  the  analysis  of  these  images  by  allowing  users  to 
digitally  generate  a  database  containing  information  about  organisms  and  seafloor 
substrate  characteristics  within  the  framework  of  a  geographic  coordinate  system.  This 
document  explains  the  input  format  requirements  and  the  GUI  functionality,  as  well  as  an 
overview  of  plans  for  future  development. 


Figure  1 :  SeaBED  AUV 
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FISHROCK  is  executed  through  Matlab  and  requires  that  three  files 
(FISHROCK.m,  FISHROCK.fig,  and  HabitatClassification.png)  reside  within  the 
Matlab  path.  The  output  of  FISH  ROCK  includes  a  comma  delimited  text  file  (*.fct)  for 
each  image  that  contains  the  navigational  information  for  the  image  and  a  list  of  the 
organisms  identified,  as  well  as  comments  noted  by  the  user.  In  addition,  upon  closing 
FISH  ROCK,  all  of  the  *.fct  files  contained  within  the  working  directory  are  assembled 
into  a  master  comma  delimited  text  file  that  can  be  loaded  directly  into  a  variety  of  other 
software  packages  (i.e.  GIS,  spreadsheet,  statistical  software)  for  further  analysis. 

Organisms  can  be  identified  and  counted  based  on  pull-down  menus  with 
predefined  lists  of  organisms  commonly  identified  in  the  study  area.  There  is  also  an 
option  to  input  the  names  of  organisms  not  included  in  the  lists.  Measurement  tools  are 
included  in  the  GUI  to  enable  the  quantification  of  the  dimensions  (length  and  area)  of 
organisms  and  objects  on  the  seafloor.  Note  that  no  corrections  have  been  made  for 
image  distortion,  and  the  GUI  assigns  a  centimeter  scale  to  each  image  based  on  the 
assumption  of  a  flat  seafloor.  Finally,  pull-down  menus  are  provided  to  allow  the  user  to 
characterize  the  substrate  type  following  a  binary  classification  scheme  that  has  been 
used  in  fish  habitat  studies  (Hixon  et  al.  1991;  Stein  et  al.  1992); 
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2.0  Data  Input 

2. 1  Image  input 

FISHROCK  is  designed  to  work  with  images  named  with  a  time  stamp  and 
image  number.  The  time  stamp  functions  as  a  unique  identifier  for  each  image  and  is 
used  to  extract  position  and  attitude  data  for  each  image.  The  required  filename  format 
includes  4  characters  at  the  beginning  of  the  file  name  followed  by  date  (yyyymmdd), 
time  (hhmmssss),  frame  number  (nnnn),  and  file  extension  (*tif,  *.jpg,  or  *.bmp): 

XXXX. yyyymmdd . hhmmssss . nnnn . j  pg 


Note  that  in  the  time  portion  of  the  filename,  seconds  are  given  as  decimal  seconds  but  no 
decimal  point  is  included  (e.g.  ssss  =  decimal  seconds:  ss.ss).  It  is  recommended  that  the 
input  data  directory  contain  only  the  images  that  will  be  analyzed  and  the  appropriate 
navigational  file  (see  section  2.2). 


In  order  to  measure  lengths  and  areas  of  organisms  on  the  seafloor  using 
FISH  ROCK  it  is  critical  that  the  look  angles  of  the  camera  be  accurately  known.  Since 
this  GUI  was  designed  for  a  specific  pixelfly  camera  the  look  angles  are  hard-coded 
within  the  GUI.  For  the  camera  used  on  SeaBED,  the  look  angles  are  42.321 1°  and 
34.26°.  These  angles  are  used  with  the  number  of  pixels  per  image  (1280  x  1024)  and  the 
altitude  of  the  vehicle  to  calculate  the  horizontal  and  vertical  dimensions  of  the  images  in 
centimeters.  Using  FISHROCKwith  images  collected  with  a  different  camera  may 
require  that  these  look  angles  be  modified.  This  modification  can  be  made  on  lines  869 
and  870  of  FISHROCK.m  (Figure  2). 


3  Edfcof  C: \mfjg>  Vtt>ti_ti  wifi  t 


File  Edft 

D  g£  b 


Text  Cel  Tods  Debug  Desktop  Whdow  Help 

U  e  ■  «  H/.oe  '  i  jj 


JOjxJ 

*  p  x 

•PZ3 


864  3tr2num  ( (char  (pxlf  lylist.  ( : ,  19  :Z0)  )  cepm^J 

865  -  auv_datenum=datenum (str2num (auvdata .adate ( : 

866  st t2num  (auvdata .adate ( : , 9 : 10) 1 , stc2num ( 

867  3tc2num (auvdata .  atime { :  ,  4 : 5} ) , stt2num (a 

960  ■:  f  ■  .  -  I 

069  -  FOV_HORIZONTAL  =  4Z  .32U*pi/18Q ; 

070  -  FOV_VERTICAL  =  34 . 26*pi/ 180  ; 

871-  auvdata -pxlflyhociz  =  2*auvdata .altitude*t _ i 

072“  auvdata .pxlf ly_vect  =  2*auvdata.altitude*ta 
073  -  navxndx= [ ) ; 

074  -  G  ::  pp=l :  size  (pxlf  ly_datenura(  1)  . 

Jj _  F  JJ 

FISH  ROCK  /narv2pxtflv  ILn  871  Col  1  j 


Figure  2:  Setting  look  angles  in  FISH  ROCK.  Lines  869  and  870  of  FISH  ROCK.m 
define  the  look  angles  of  the  pixelfly  camera  used  on  the  SeaBED  AUV.  If  using 
this  code  with  data  collected  with  a  different  camera,  these  look  angles  must  be 
correct  in  order  to  calculate  the  scale  of  the  image. 
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2.2  Navigation  data  input 

2.2. 1  Use  of  the  catjiav.mat  file  generated  for  SeaBED 

FISHROCK  seeks  a  binary  matlab  data  file  (*.mat  file)  called  catnav.mat  as 
primary  navigational  input.  This  file  should  reside  in  the  same  directory  as  the  images 
that  will  be  analyzed,  and  is  created  as  a  standard  output  product  in  the  SeaBED  AUV 
data  processing  pipeline.  The  first  time  the  code  is  initialized  within  a  particular  data 
directory,  cat  nav.mat  is  used  in  conjunction  with  a  listing  of  all  image  files  in  that 
directory  to  create  a  subsampled  navigation  file  that  is  used  by  FISH  ROCK  to  index 
navigation  data  to  the  images.  Another  mat  file  called  pxlflyNAV.mat  is  automatically 
generated  based  on  the  time  stamps  of  the  images,  and  is  used  directly  by  FISHROCK 
during  subsequent  use  of  the  code.  In  addition,  a  text  file,  pxlflyNAV.txt  (Figure  3), 
which  contains  the  same  information  as  pxlflyNAV.mat  will  also  be  generated,  and  is 
provided  as  a  resource  to  users.  The  format  of  this  file  is: 

image_name,  latitude,  longitude,  water  depth  (m) ,  altitude  (m) ,  horiz 
image  dim  (m) ,  vert  image  dim  (m) ,  total  image  area  (mA2) ,  local  x  (m) , 
local  Y  (m) 


fw  nun.  pmn 


ac]»i 


Figure  3:  Example  pxlflyNAV.txt  file  that  is  generated  by  FISH  ROCK  containing 
navigational  information  for  each  image  in  the  directory. 


2.2.2  Creating  a  catjiav.mat  file 

If  using  FISH  ROCK  with  different  input  data,  a  cat  nav.mat  file  can  be  created 
within  Matlab,  and  must  contain  a  data  structure  called  auvdata.  This  data  structure  must 
contain  the  following  fields: 
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Table  1 :  Fields  that  must  be  included  in  cat  nav.mat 


Field 

Size 

type 

format 

adate 

N  x  10 

char 

yyyy/mm/dd 

atime 

N  x  11 

char 

hh:mm:ss.ss 

localX 

N  x  1 

double 

localX  or  UTMX  (m) 

localY 

N  x  1 

double 

localY  or  UTMY  (m) 

altitude 

Nx  1 

double 

altitude  of  camera  (m) 

depth2 

Nx  1 

double 

total  water  depth  (m) 

lat 

N  x  1 

double 

decimal  degrees  (-S) 

Ion 

N  x  1 

double 

decimal  degrees  (-W) 

2.2.3  Using  a  text  file  for  nav  input 

If  neither  catnav.mat  nor  pxlflyNAV.mat  exist,  the  user  is  prompted  to  select  a 
text  file  as  navigational  input.  This  functionality  provides  the  option  of  using 
F1SHROCK  with  data  not  collected  with  SeaBED.  Remember,  if  using  a  different 
camera  system  with  different  look  angles,  they  must  be  declared  within  FlSHROCK.m 
(Figure  2;  section  2.1).  Once  chosen,  the  text  file  will  be  uploaded  and  used  to  generate 
the  cat  nav.mat  file  as  well  as  pxlflyNAV.mat  and  pxlflyNAV.txt  (see  section  2.2.1). 

This  text  file  must  be  tab  delimited,  with  no  header  (Figure  4),  following  the  format: 

date  time  localX  localY  altitude  depth  lat  Ion 

The  date  must  be  provided  with  the  format  yyyy/mm/dd,  and  time  must  be  hh:mm:ss.ss 
(Fig.  4).  If  not  working  in  localXY,  any  values,  including  UTM  coordinates  or  a  number 
indicating  no  data  (i.e.  9999)  may  be  used  to  populate  the  localX  and  localY  fields.  The 
precision  of  the  latitude  and  longitude  coordinates  and  attitude  data  should  be  determined 
by  the  user  based  on  the  dataset  being  analyzed. 
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Figure  4:  Example  of  text  file  used  as  navigational  input. 
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3.0  Using  FlSH_ROCK 

Before  FISHROCK  can  be  executed,  the  three  required  files  (FISHROCK.m, 
FISHROCK.fig,  and  HabitatClassification.png)  must  be  copied  to  the  hard  drive.  After 
opening  Matlab,  the  path  must  be  modified  to  include  the  folder  that  contains  these  files. 
Once  this  is  done,  the  GUI  is  initialized  by  typing  FISH  ROCK  in  the  Matlab  command 
window  (Figure  5). 


CHOOSE  START  IMAGE 

ADVANCE  TO  NEXT  IMAGE 
IMAGE  DATA 

DATE: 

TIME: 

LATITUDE: 

LONGITUDE: 

WATER  DEPTH: 

ALTITUDE: 

IMAGE  AREA: 


DATABASE  TOOLS 


ROCKFISH 

Unknown  ronkFsh 

COUNT 

1  r  FLATFISH 

| unknown  flatfish 

j 

C  SKATES 

[unkrown  skats 

J 

r  ROUNDFISH 

Unkrrjwn  roundish 

J 

RESET 

r  INVERTS 

unkt  uwn  1'iverlebiale 

J 

r  OTHER 

! 

ENABLE  PAN 


MEASUREMENT  TOOLS 


SAVE  ANNOTATED  TIE 


SU8STRATE  TYPE 


[unknown  [unknown  pj  Scheme  Wo  I 


COMMENT  P 


FISH_ROCK  version  1.1 
vf#mni@whoi  edu 


Figure  5:  FISH  ROCK  Graphical  User  Interface  (GUI)  as  it  appears  on  startup. 


3. 1  Choose  Start  Image 

To  begin  using  FISH  ROCK,  first  select  an  image  to  start  with  by  pressing  the 
Choose  Start  Image  button  located  in  the  upper  right  corner  of  the  GUI  (Figure  5).  The 
user  will  then  be  prompted  to  browse  through  their  computer  to  select  an  image  (Figure 
6).  Once  an  image  has  been  selected,  the  program  will  gather  the  necessary  navigational 
information  from  the  navigation  file  (see  section  2)  which  must  reside  in  the  same 
directory  as  the  images.  The  selected  image  and  its  corresponding  navigational 
information  will  then  be  displayed  in  the  GUI,  and  the  Choose  Start  Image  button  will  no 
longer  be  visible  (Figure  7). 
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Figure  6:  Choose  Start  Image  browsing  tool. 
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Figure  7:  FISH  ROCK  as  it  appears  after  choosing  the  first  image  to  be  analyzed.  Note 
that  the  Choose  Start  Image  button  is  no  longer  visible. 


3. 2  Nav  Display  -  Image  Data 

Upon  loading  an  image,  its  navigational  data  is  displayed  on  the  right  side  of  the 
GUI  window  to  help  orient  the  user  within  the  geographic  framework  of  the  dataset 
(Figure  7).  The  navigation  information  that  is  displayed  includes  date,  time,  lat,  long, 
water  depth,  altitude,  and  image  area  (assuming  a  flat  seafloor)  and  is  all  derived  from  the 
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navigation  file  (see  section  2).  The  image  name,  which  also  contains  date  and  time,  is 
displayed  in  the  GUI,  above  the  image  itself  (Figure  7). 


3. 3  Navigating  Through  a  Dataset 

After  selecting  the  first  image,  the  user  can  navigate  through  the  images  in  the 
folder  using  the  Advance  to  Next  Image  button  (Figure  7).  After  advancing  to  the  second 
image,  a  Previous  Image  button  will  appear  in  the  upper  right  comer  of  the  GUI  to  allow 
the  user  to  step  back  to  the  previous  image  (Figure  8). 
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Figure  8:  FISHROCK  as  it  appears  after  advancing  to  the  second  image  in  the  dataset. 
The  Previous  Image  button,  now  visible  in  the  upper  right  comer  of  the  GUI,  can  be 
used  to  step  back  to  the  previous  image,  and  the  Advance  to  Next  Image  button  can 
be  used  to  step  forward. 


If  FISH  ROCK  has  been  used  during  a  prior  iteration  to  identify  organisms  or 
substrate  or  to  track  comments,  this  information  will  be  automatically  displayed  in  the 
GUI  when  navigating  through  the  dataset.  Substrate  Type  and  Comments  can  be  directly 
modified  within  FISH  ROCK.  To  change  organisms  that  have  already  been  identified, 
however,  the  user  must  either  modify  the  *fct  file  or  use  the  Reset  button  to  reset  all  data 
for  the  image  to  the  default  values  (see  section  3.4.1). 

If  organisms  have  been  identified  and  counted,  substrate  has  been  identified,  or  a 
comment  has  been  entered  for  a  particular  image,  or  if  a  data  is  uploaded  from  an  existing 
*fct  file,  pressing  Advance  to  Next  Image  or  Previous  Image  button  will  prompt  the  user 
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to  declare  whether  the  information  gathered  on  the  current  image  should  be  saved  (Figure 
9).  Saying  yes  will  create  an  *fct  file  for  this  image  containing  the  information  that 
appears  in  the  GUI,  and  will  overwrite  any  existing  *fct  file  for  this  image  (see  section  4). 


->  ME 

E  -=1iJ-x| 

Do  you  w; 

YES[J 

ant  to  save?: 

J 

NO 

Figure  9:  Menu  window  that  appears 
to  confirm  whether  data  should  be 
saved  in  an  *  fct  file. 


3.4  Database  Tools 

3.4.1  Selecting  Organism  Type/Name 

Organisms  may  be  counted  and  identified  by  using  the  Database  Tools,  located  in 
the  lower  left  corner  of  the  GUI  (Figure  10).  The  radio  buttons  can  be  used  to  select  the 
organism  category  (e.g.  rockfish,  flatfish,  skates,  roundfish,  inverts)  and  more  specific 
identifications  can  be  made  using  the  pull-down  menus  (to  species  on  some  cases).  The 
Other  category  can  be  used  to  identify  organisms  that  are  not  included  in  the  predefined 
lists  or  to  log  additional  data  that  may  be  important  to  include  in  the  database.  For 
example,  if  anthropogenic  material  (fishing  nets,  lobster  pots,  miscellaneous  debris,  etc) 
is  visible  in  a  particular  image,  the  other  option  provides  a  means  for  logging  this 
information  within  the  database. 
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Figure  10:  Database  Tools  portion  of  FISFtROCK  GUI. 


3.4.2  Counting  Organisms 

Pressing  COUNT  after  selecting  a  radio  button  or  pull-down  menu  item  enables 
the  counting  function  and  allows  the  user  to  left  click  on  organisms  in  the  image  to 
identify  them.  Note  that  left-clicking  when  the  count  function  is  engaged  indicates  that 
additional  organisms  of  this  type  will  be  identified  at  this  time.  Right-clicking,  however, 
will  disengage  the  counting  function  after  identifying  a  single  organism.  For  example,  if 
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there  arc  three  unknown  rockfish  in  the  image,  the  user  should  first  select  the  ROCKFISH 
radio  button,  and  then  press  COUNT.  The  first  two  organisms  should  then  be  identified 
using  the  left  mouse  button,  and  the  third  organism  should  be  selected  with  the  right 
mouse  button.  If  only  one  rockfish  is  to  be  identified,  the  user  must  select  COUNT  and 
then  use  the  right  mouse  button  to  identify  that  organism. 

3.4. 1  Annotation  of  Identified  Organisms 

When  identifying  organisms,  a  yellow  and  black  cross-hair  symbol  is  displayed 
on  the  image  to  mark  the  organism’s  location.  In  addition,  a  symbol  indicating  the  fish 
ty  pe  and  index  will  be  displayed  beside  the  cross-hair  symbol  referring  to  the  broad 
category  chosen  (Rockfish:  RO#;  Flatfish:  FL#;  Skates:  SK#;  Roundfish:  RN#;  Inverts: 
IN#;  Other:  OT#),  and  the  cumulative  index  of  organisms  in  this  category  (Figure  1 1). 

For  example,  if  3  rockfish  are  chosen,  there  will  be  3  symbols  (RO  1 ,  R02,  and  R03) 
displayed  on  the  image  (Figure  1 1). 
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Figure  1  1 :  FISH  ROCK  with  three  rockfish  identified  and  marked  with  cross-hairs 


3.4. 1  Resetting  data 

The  RESET  button  included  in  the  DATABASE  TOOLS  portion  of  the  GUI 
(Figure  10)  can  be  used  to  purge  all  information  for  the  current  image.  This  includes  not 
only  organisms,  but  also  substrate  type,  comments,  and  measurements.  If  an  output  data 
file  (*.fct)  was  created  for  this  image  during  previous  usage  of  the  GUI,  the  *.fct  file  will 
be  deleted  upon  resetting  the  database  information. 
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3.5  Measurement  Tools 

The  measurement  tools  provided  in  FISHROCK  assume  a  flat  seafloor  and  that 
the  organisms/objects  being  measured  are  on  the  seafloor.  The  distance  of  the  seafloor 
from  the  camera  lens  is  assumed  to  be  equal  to  the  vehicle  altitude.  Both  length  and  area 
measurements  are  displayed  in  the  GUI  directly  above  the  buttons  used  to  enable  the 
measurement  tools.  Measurements  can  also  be  assigned  to  a  particular  organism/object 
that  has  been  counted.  If  a  measurement  has  been  assigned  to  an  organism,  the  cross-hair 
symbol  identifying  the  location  of  the  organism  will  change  to  indicate  which 
measurement  has  been  assigned  (see  sections  3.5.1  and  3.5.2). 

-  MEASUREMENT  TOOLS- 


LENGTH  AREA 


Figure  12:  Measurement  Tools  portion  of  FISH  ROCK  GUI. 


3.5.1  Length  Measurement 

To  measure  the  length  of  an  organism  or  object,  press  on  LENGTH,  and  left-click 
on  a  point  in  the  image  to  begin  the  line.  This  point  will  be  annotated  with  red  cross-hair 
symbol,  and  a  rubber-band  tool  with  a  thick  blue  line  will  be  enabled  to  visually  aid  the 
user  in  identifying  the  position  and  orientation  of  the  line  being  drawn  (Figure  13).  To 
end  the  line,  the  user  must  right-click  on  the  last  point  of  the  line.  This  function  can  also 
be  used  to  measure  a  polyline,  by  left-clicking  on  all  points  except  for  the  last  one,  which 
must  be  identified  by  right-clicking  to  disengage  the  measurement  tool. 
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Figure  13:  Example  of  use  of  Length  measurement  tool  of  FISHROCK  GUI  with  the 
rubber-band  tool  displayed  on  the  portion  of  the  image  that  is  measured.  In  this 
example,  the  fish  being  measured  has  already  been  identified  as  R03. 


Once  the  line  is  completed,  and  the  measuring  tool  is  disengaged,  the  length  is 
displayed  above  the  Length  tool  button  (Figure  13).  In  addition,  the  user  is  prompted  to 
declare  whether  or  not  the  measurement  should  be  assigned  to  an  organism  that  has  been 
identified  (Figure  14a).  To  avoid  over-populating  the  image  with  lines  and  symbols,  the 
blue  line  and  red  cross-hair  symbols  are  removed  from  the  image  after  this  choice  is 
made.  If  Yes  is  selected,  a  second  menu  will  appear  displaying  a  list  of  all  organisms 
currently  identified  in  the  image  (Figure  14b).  This  menu  is  used  to  select  the  organism 
to  which  the  measurement  should  be  assigned.  If  a  length  measurement  is  assigned  to  an 
organism  that  was  identified  through  the  GUI,  the  cross-hair  symbol  associated  with  that 
organism  will  be  surrounded  by  a  yellow  square  (Figure  15). 
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Figure  14a:  Assign  length  to  organisms  menu 
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Figure  14b:  List  of  organisms  that  have  been  identified  in  the 
image  used  to  assign  measurement  to  an  organism. 


.zIsJjlJ 


CCXC.20051 01 8.14541 336.01 27.jpg 


20  40  60 

DATABASE  TOOLS 


ROCKFISH 

[unfcnovvn  focktisb 

r  flatfish 

|unkriuwn  Hal  fish 

J 

r  SKATES 

[unknown  gvata 

J 

r  ROUNDTISt: 

Jurtmnwr  ‘niinrtfsh 

-ll 

r  INVERTS 

[unknown  rvwisbrats 

”3 

140  160  180 

MEASUREMENT  TOOLS 
LENGTH 
114  cm 

LENGTH  I 


SAVE  ANNOTATED  T1F 


SUBSTRATE  TVPE 

[unknown  [unknown  Scheme  Info 


COMMENT  ‘ 


FlSH_ROCK  version  1  1 
Vemndjjjjwhoi  *<fu 


Figure  15:  FISH  ROCK  after  assigning  a  length  measurement  (1 1 .4  cm)  to  an  organism 
(R03).  The  yellow  square  surrounding  the  cross-hair  symbol  indicates  that  a  length 
measurement  has  been  assigned  to  this  organism. 


3.5.2  Area  Measurement 

The  area  measurement  tool  is  operationally  similar  to  the  length  measurement 
tool.  To  measure  the  area  of  an  organism  or  object,  first  press  on  AREA ,  then  left-click  to 
identify  each  of  the  points  in  the  polygon  except  the  last  point  which  must  be  right- 
clicked.  Each  point  of  the  polygon  will  be  marked  by  red  cross-hair  symbols  connected 
by  a  blue  line.  As  with  the  length  measurement  tool,  a  rubber-band  tool  is  displayed  with 
a  blue  line  to  help  identify  the  correct  placement  of  points  of  the  polygon  on  the  image 
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(Figure  1 6).  After  closing  the  polygon,  the  area  measurement  is  displayed  above  the 
AREA  tool  button  (Figure  16).  As  with  the  length  measurement  tool,  the  user  will  then  be 
prompted  to  declare  whether  or  not  to  assign  the  measured  area  to  an  organism  that  has 
been  counted  (Figure  14).  If  an  area  measurement  is  assigned  to  an  organism  that  was 
identified  through  the  GUI,  the  cross-hair  symbol  associated  with  that  organism  will  be 
surrounded  by  a  black  circle  (Figure  17).  Note  that  if  both  length  and  area  measurements 
are  assigned  to  a  single  organism,  the  cross-hair  symbol  that  identifies  it  will  be 
surrounded  by  both  a  yellow  square  (Figure  15)  and  a  black  circle  (Figure  17). 
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Figure  1 6:  Use  of  area  measurement  tool  to  measure  the  area  of  an  invertebrate  (IN  1 ). 
Once  the  last  point  of  the  polygon  has  been  selected  with  the  right  mouse  button,  the 
measurement  is  displayed  above  the  AREA  button,  and  the  user  is  prompted  to 
declare  whether  or  not  this  measurement  should  be  assigned  to  an  organism  (Figure 
17). 
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Figure  17:  After  assigning  the  area  measurement  (42.7  cm2)  to  an  organism  (INI)  the 
measurement  polygon  is  removed  from  the  image  and  the  cross  hair  symbol  that 
identifies  the  location  of  the  organism  is  surrounded  by  a  black  circle. 


3. 6  Substrate  Type 

Substrate  type  is  defined  by  using  the  two  pull-down  menus  under  Substrate  Type 
near  the  bottom  of  the  GUT  (Figure  18).  A  figure  explaining  the  binary  habitat 
classification  scheme  (Figure  19)  can  be  viewed  by  pressing  the  Scheme  Info  button 
located  within  the  Substrate  Type  portion  of  the  GUI  display  (Figure  1 8).  The  binary 
classification  scheme  requires  that  two  substrate  types  (primary  and  secondary)  be 
selected  with  the  pull-down  menus  in  the  GUI  (Hixon  etal.,  1991;  Stein  et  al.,  1992). 
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Figure  18:  Substrate  Type  tool. 
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Binary  Habitat  Classification  System 
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Figure  19:  Benthic  Habitat  Binary  Classification  Scheme 


3. 7  Other  Tools 
3.7.1  Grid  On/Off 

The  Grid  On  button  is  located  on  the  right  side  of  the  GUI,  just  below  the 
navigational  information  display  (Figure  17)  and  is  used  to  overlay  a  grid  on  the  current 
image.  The  grid  is  based  on  the  calculated  footprint  of  the  image  (assuming  flat  seafloor) 
based  on  the  vehicle  altitude  and  the  field  of  view  of  the  camera  (section  2.1).  This 
button  toggles  to  Grid  Off  after  it  is  enabled  (Figure  20),  and  can  be  pressed  to  remove 
the  displayed  grid  from  the  view. 
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Figure  20:  When  Grid  On  is  pressed,  a  grid  is  displayed  on  the  image,  and  the  Grid  On 
button  toggles  to  Grid  Off. 


3. 7. 1  Zoom  In/Out 

The  Zoom  buttons  are  located  on  the  right  side  of  the  GUI,  just  below  the 
navigational  information  display  (Figure  20)  and  are  used  to  zoom  the  view  in  or  out. 
Note  that  if  grid  lines  are  displayed,  they  will  scale  when  the  zoom  buttons  are  used. 


3. 7.2  Enable/Disable  Pan 

The  Enable  Pan  button  is  located  on  the  right  side  of  the  GUI,  just  below  the 
navigational  information  display  (Figure  20),  and  can  be  used  to  change  the  current  view 
within  the  image.  Clicking  on  it  will  toggle  the  button  to  Disable  Pan,  which  should  be 
pressed  when  finished  navigating  around  the  image.  To  Pan,  click  on  the  Enable  Pan 
button,  then  left-click  on  the  image,  hold,  and  drag  the  image  to  the  desired  view. 


3. 7.4  Save  Annotated  Tif 

The  Save  Annotated  Tz/button  is  used  to  generate  a  full-resolution  tiff  file  of  the 
current  image  with  the  displayed  annotation  and  a  centimeter  scale  along  each  axis 
(Figure  21).  If  a  grid  is  overlain  on  the  image  when  Save  Annotated  Tif  is  pressed,  the 
resulting  tiff  image  will  include  those  grid  lines  (Figure  21b).  If  the  current  view  is 
zoomed,  however,  it  is  important  to  note  that  the  output  tiff  File  will  be  saved  with  the  full 
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view  rather  than  the  zoomed  view.  The  output  image  has  the  same  root  name  as  the 
input  image,  and  has  cm  scales  added  to  both  X  and  Y  axes. 
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Figure  21a:  Example  of  annotated  image  in  FISH  ROCK. 
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CCXC.20051018.14541336.0127.jpg 
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cm 

Figure  2 1  b:  Example  of  annotated  tif  image  created  from  image  displayed  in 

FISH  ROCK  in  Figure  21a  (CCXC.20051018.l4541336.0127_annotated.tif). 

Note  that  this  image  was  created  with  a  grid  overlain  on  the  display. 


3. 9  Closing  the  GUI 

Upon  exiting  the  GUI  with  the  Close  button,  a  COMBINEDFCT.fct  file  is 
created  by  concatenating  all  information  in  the  {image_name}.fct  Files  (see  section  4). 
The  COMBINED  FCT.fct  file  is  a  self-updating  text  file  that  includes  data  assembled 
from  all  *.fct  Files  in  the  working  directory. 

When  the  user  reaches  the  last  image  in  the  directory  that  is  associated  with 
navigational  data,  it  will  display  a  message  indicating  that  there  are  no  more  Files.  It  will 
then  automatically  perform  the  actions  that  occur  when  CLOSE  is  pressed. 
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4.0  Database  Output  -  FCT  Philosophy 

For  each  image  file,  an  *.fct  (Fish  CounT)  file  is  created  with  the  same  root  as  the 
input  image.  For  example,  for  image  CCXC.20051018.14541036.0126.jpg,  a  text  file 
will  be  created  called  CCXC.20051018.14541036.0126.fct.  If  data  have  been  assembled 
for  a  particular  image,  the  user  is  prompted  to  declare  whether  or  not  to  save  the  data  (see 
section  3.3).  Saying  yes  will  overwrite  any  existing  *.fct  file  for  that  image.  Note  that  if 
an  *fct  file  exists  for  an  image  that  is  being  displayed  in  the  GUI,  the  information 
contained  within  it  will  be  displayed  when  advancing  through  the  images  (see  section 
3.3). 


Each  *.fct  file  contains  a  comma  delimited  list  of  each  of  the  organisms  that  were 
counted/identified  as  well  as  navigation  and  image  information.  In  order  to  make 
F1SH  ROCK  a  versatile  tool  that  can  be  used  to  address  a  suite  of  scientific  questions, 
each  line  in  the  text  file  contains  navigational  data,  fish  identifications  and  measurements, 
as  well  as  substrate  type  and  comments.  In  addition,  the  location  of  each  organism  within 
the  image  is  provided,  which  after  further  software  development,  can  be  used  to  address 
questions  associated  with  spatial  distribution  on  a  fine  scale. 

FCT  file  format: 

lat, long, water  depth, image  name , date , t'i me , image  area, image 
width, image  hei ght , substrate , fi sh  type, fish 
subtype , i ndex , posx , posY, organi sm  length, organism 
area , comment 

FCT  default  values: 

substrate  type:  not  classified 
organism  length:  Nan 
organism  area:  Nan 
comment:  no  comment 
fish  type:  Nan 
fish  subtype:  Nan 


Upon  closing  the  GUI,  the  COMBINEDFCT.fct  file  is  created  using  information 
in  all  of  the  individual  *.fct  files.  Since  this  file  is  regenerated  each  time  the  GUI  is 
closed  with  the  Close  button,  it  is  self-updating  based  on  the  *.fct  files  in  the  directory. 
This  file  can  be  loaded  directly  into  a  GIS  package,  a  spreadsheet,  or  statistical  software 
for  additional  analysis. 
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5.0  Conclusions  and  Plans  for  Future  Development 

FISHROCK  is  a  tool  intended  to  create  digital  databases  for  the  analysis  of 
information  contained  within  bottom  photographs.  The  development  of  FISH  ROCK  is 
an  important  step  toward  efficiently  utilizing  large  datasets  of  digital  photographs  to 
address  a  range  of  oceanographic  problems.  Data  files  created  with  FISH  ROCK  are  text 
files  that  can  be  directly  uploaded  into  statistical  software,  a  spreadsheet,  or  a  GIS 
package.  Although  the  current  version  is  specific  to  benthic  organisms  common  off  the 
west  coast  of  the  U.S.,  it  can  be  modified  for  use  in  a  variety  of  environments  including 
the  deep  sea.  The  pull-down  menus  can  be  customized  so  the  GUI  could  be  used  to 
assess  not  only  benthic  organism  abundance  and  distribution,  but  also  geologic  features, 
and  the  distribution  of  archeological  artifacts. 

Creating  additional  modules  that  can  be  used  in  conjunction  with  FISH  ROCK 
will  facilitate  more  advanced  analysis.  For  example,  using  the  information  already 
contained  within  the  *.fct  files,  a  module  can  be  written  to  quantitatively  analyze  the 
distribution  of  organisms/objects  at  a  variety  of  spatial  scales.  In  addition,  by  utilizing 
image  feature  matching  algorithms  already  developed  at  WHOI/DSL  for  photo- 
mosaicking  applications  (Pizarro  and  Singh,  2003;  Figure  22)  in  conjunction  with  data  in 
the  *.fct  fdes  an  additional  module  can  be  created  to  automate  the  removal  of  redundant 
data  from  transects  of  images,  automatically  providing  accurate  estimates  of  abundance 
and  diversity  per  unit  area.  The  feature  matching  algorithm  can  also  be  used  to  measure 
organisms/objects  that  are  in  the  water  column  between  the  camera  and  the  seafloor  as 
long  as  the  object  is  imaged  in  subsequent  photos.  Finally,  future  work  will  also  include 
the  application  of  distortion  corrections  to  improve  the  accuracy  of  length  and  area 
measurements. 
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Figure  22:  Photo-mosaic  of  the  three  adjacent  images  presented  in  this  report.  The 
feature-matching  algorithm  used  for  this  mosaic  is  a  robust  tool  that  can  be  used  to 
enhance  the  analytical  functionality  of  FISH  ROCK  in  future  versions  of  the  code. 
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